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Assessing the Gains from Concept Mapping in Introductory Statistics 


Abstract 

In an effort to improve active learning in introductory statistics, we introduce the use of concept mapping 
techniques as part of the course. While previous papers have touted the use of this and other interactive 
teaching methods in statistics education, we add to this literature by providing additional assessment of its 
efficacy. This comes through an experimental design that involves a single instructor teaching two sections of 
the same statistics course over the same semester. Both cover the same material in the same way with the 
exception that concept mapping is used in one section, but not the other. Assessment of learning outcomes is 
done through the use of pre-tests and post-tests of understanding of statistical concepts. We also track changes 
in student s study habits over the semester through additional surveys. We find only weak evidence that 
concept mapping is effective in aiding student learning of statistics. 
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Abstract 

In an effort to improve active learning in introductory statistics, we introduce the use of 
concept mapping techniques as part of the course. While previous papers have touted the 
use of this and other interactive teaching methods in statistics education, we add to this 
literature by providing additional assessment of its efficacy. This comes through an 
experimental design that involves a single instructor teaching two sections of the same 
statistics course over the same semester. Both cover the same material in the same way 
with the exception that concept mapping is used in one section, but not the other. 
Assessment of learning outcomes is done through the use of pre-tests and post-tests of 
understanding of statistical concepts. We also track changes in student's study habits over 
the semester through additional surveys. We find only weak evidence that concept 
mapping is effective in aiding student learning of statistics. 


Introduction 

Across the Academy we have heard the call to move away from the "chalk and talk" 
approach to teaching and adopt a pedagogy that engages students more in the learning 
process. We have attended workshops on collaborative learning and been introduced to 
activities such as classroom experiments and one-minute papers. We have read the 
journal articles describing teaching innovations and have been told that these activities 
enhance student engagement and increase student satisfaction with our courses. Yet 
relatively few of these papers evaluate the effect of these activities on objective measures 
of student learning. Indeed, Holleran, Taylor, and Santopietro (2006) report the results of 
their examination of the contents of the Journal of Economic Education and the Journal of 
Economics and Finance Education for the period of 2001-2006. They note that of nearly 
two dozen articles describing a specific active-learning technique none reported any 
empirical evidence of the effect of that activity on student learning. 

One goal of this paper is to address this concern by assessing empirically the effect of 
introducing a particular teaching innovation into the classroom. Specifically we describe 
the use of Concept Maps in an introductory statistics course and report the results of 
testing the marginal effect of this exercise on various measures of student performance 
and study-related behavior. This process of introducing, assessing and, where 
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appropriate, revising teaching and learning activities is part of what can be considered 
classroom research. 

Kochis (2006) defines classroom research as "the systematic investigation of the effects of 
our teaching on student learning for the purpose of improving instruction". These 
investigations produce feedback about student learning to instructors that is typically used 
to inform teaching decisions made by instructors in their classrooms. One aspect of 
classroom research is the investigation of patterns of learning in a particular course - what 
are the barriers to learning, what teaching strategies can improve performance, how 
effective are the new strategies employed. This paper attempts to answer such questions 
in the context of a required introductory statistics course taken by all business and 
economics students at the University of Minnesota Duluth. 

delMars, Garfield and Chance (1999) developed a model of classroom research for 
statistics education structured on the basis of four questions: 

• What is the problem? 

• What technique can be used to address the learning problem? 

• What type of evidence can be gathered to show whether the implementation 
is effective? 

• What should be done next, based on what was learned? 

These questions outline both the approach taken in this study and the presentation of 
results. In the first section we discuss the problem encountered by our students, 
particularly their failure to understand the connection between key concepts in statistics. 
Following that we describe the introduction of concept mapping as a technique to address 
this learning problem. Next we describe our assessment of the effectiveness of concept 
maps on student outcomes and finally we make suggestions about how we might refine 
the activity in the future. 


What is the Problem? 

A common theme that runs through the statistics education literature is that most students 
in introductory statistics courses lack an understanding of the relationship among 
important concepts in statistics (e.g., Garfield, 1994; Schau and Mattern, 1997; delMas, 
Garfield, and Chance, 1999). Yet a vital dimension of the statistical expertise needed 
statistical reasoning and problem solving is the "presence of an understanding of the key 
elements of statistics and their relationships" (Turns eta/, 2000). 

As instructors we view the introductory statistics course as a consistent story of related 
concepts as we move from description statistics through probability to statistical inference. 
Unfortunately students often miss the big picture and see statistics as a series of 
disconnected topics. Students may be able to calculate a standard deviation and a 
standard error but often they do not understand how these concepts are related (both in 
terms of similarities and differences). As a result, students often confuse the concepts 
using one when they should have used the other. We particularly see the consequences of 

2 


https://doi.org/! 0.20429/ijsotl.2007.010219 


2 



IJ-SoTL, Vol. l [2007], No. 2, Art. 19 


this lack of understanding in the area of statistical inference as students fail to make the 
connections between fundamental concepts, such as sample, population, sampling 
distributions, and sampling variability. 

Because of this isolated understanding of concepts, students remain novices in their 
statistical thinking (Schau and Matten, 1997). Students tend to memorize, for example, 
the steps of estimation and hypothesis testing without truly understanding the process. As 
we introduce different situations (difference between means, difference between 
proportions, hypothesis testing in regression, etc.) students become overwhelmed thinking 
of each of these as separate topics rather than a part of an integrated whole. This failure 
to develop a deep understanding of the integration of concepts affects their clarity of 
learning and retention of knowledge. Ausubel (1968) advanced a theory which 
contrasted meaningful learning from rote learning. One characteristic of rote learning is the 
lack of a deliberate effort to relate new knowledge to prior learning. This is contrasted with 
the affective commitment to relate new knowledge to prior learning which occurs in 
meaningful learning. It would appear that many of our statistics students are engaged in 
rote learning rather than the meaningful learning needed for effective statistical reasoning 
and problem solving in the introductory course as well as retention of knowledge necessary 
for success in future courses in the business and economics curricula and their eventual 
careers. 

To initially identify the problem students in one section were having with the course we 
administered the GAMES® Survey Instrument developed by Marilla Svinicki (2004). The 
survey items consist of components of effective study behavior - goal-oriented study (G), 
active studying (A), meaningful and memorable studying (M), explaining the material in 
order to learn it (E), and self-monitoring (S). According to Svinicki (p. 131) students who 
engaged in the learning behaviors reflected by this model will learn at a deeper level. 
However, students self-reported that, when studying, they seldom "make connections 
between what I am studying and past classes or units" or "create outlines, concept maps, 
or organizational charts of how the ideas fit together." The results of the survey led us to 
specifically introduce concepts mapping as a class exercise to encourage students to make 
connections among the key concepts of statistical inference. 


What Technique Can Be Used to Address the Learning Problem? 

Concept maps are an increasingly popular technique used to support "meaningful learning." 
According to Ausubel (1968) "the most important single factor influencing learning is what 
the learner already knows." Based on Ausubel's principles of meaningful learning in which 
there must be a deliberate effort to relate new knowledge to prior learning, Novak and 
Gowin (1984) developed concept maps for use in science education. Concept mapping 
involves the visual representation of how concepts within a domain are interrelated. A 
concept map consists of a hierarchal structure of cells or nodes that represent concepts or 
ideas and labeled cross-links representing the relations between concepts. These cross¬ 
links between concepts can be non-, uni-, or bi-directional. The links connect not only 
adjacent concepts but concepts in different domains of the concept map as well. It is this 
feature that distinguishes concept maps from other organizers such as outlines and mind 
maps. The resulting network of concepts can be an aid to learning by explicitly integrating 
new and old knowledge and assist students in understanding the relationships between 
statistical concepts such as populations, samples, sampling 


3 


https://doi.0rg/l0.20429/ijs0tl.2007.010219 


3 



Assessing the Gains from Concept Mapping in Introductory Statistics 


distributions and statistical inference. 



Introductory statistics is an ideal course for implementing such a learning tool. The course 
typically is designed to develop groundwork for inferential statistics right from the start 
with each concept covered building upon previous concepts. The successful student must 
relate new knowledge to prior learning. Additionally, relational thinking, seeing the 
interconnectedness among concepts such as populations and samples or variability and 
sampling error, is necessary for truly and deeply understanding the material. 

In addition to creating explicit links between concepts leading to meaningful learning, the 
process of concept mapping in class may have other benefits as well. Educational and 
cognitive theorists have posited that an active, hands-on approach to course material leads 
to higher-order learning; thus the mapping process is a learning experience in and of itself. 
Concept mapping is often introduced as a small group activity (e.g., Brown, 2003; Prezler, 
2004; Calderon-Steck, 2006). The benefits of cooperative/collaborative learning have been 
well documented across academic disciplines (e.g., Giraud (statistics), 1997; Maier and 
Keenan (economics), 1994). Several studies have evaluated the usefulness of concept 
maps across three groups, a control group, individual mappers and group mappers 
(Brown, 2003; Calderon-Steck, 2006; Chiou, 2006) finding that group mappers 
outperformed those who did no mapping or who did mapping as individuals. Thus, it is 
difficult to disentangle the independent effect of the concept mapping from the effects of 
cooperative learning. 

John Budd (2003) describes an in-class exercise in which small groups of students create a 
Mind Map, which are similar to Concept Maps, for a specific topic. In addition to the 
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benefits of a collaborative learning technique such as this, he notes that an exercise such 
as this can re-energize a course. Just the process of having students work with colored 
markers and large sheets of paper increases the energy level of the class and introduces 
an opportunity for creative expression not found at other times during the course. That 
change of pace can increase student motivation resulting in the outcome, as reported in a 
number of studies, that students think that concept mapping is a useful exercise. 


Implementation 

The primary goal in introducing concept maps to students in introductory statistics is 
twofold. The course-specific goal is increasing conceptual understanding of statistical 
inference by explicitly linking course content on an on-going basis throughout the course. 
Additionally, we seek to improve student's study habits by providing an active learning tool 
that should be useful in many learning environments. One aim of the current study is to 
provide evidence as to whether or not these goals are met through the introduction of 
concept mapping into the learning process. The first step in doing this, then, is to design 
an experiment in which these desired outcomes can be measured. 

Design of Experiment 

With the same instructor teaching two sections of the same course, it was possible to 
introduce the use of concept maps into one and then treat the other section as a control 
group with which to compare outcomes. Having the same instructor for both groups 
allowed instruction in both sections to cover the same material in the same way, removing 
the potential for instructor bias in the outcomes, although there are likely additional 
sources of bias in such an experiment. 

The course in which the experiment took place was a standard introductory business 
statistics course, required for majors in business and economics. Typical content coverage 
includes organizing data into tables and charts, numerical descriptive/summary statistics, 
probability, probability distributions, sampling distributions, statistical inference, including 
estimation and hypothesis testing, simple regression and correlation analysis. The 
teaching approach was a mix of lecture, examples and worked problems, along with 
hands-on practice with data analysis using Microsoft Excel. Standard assessment tools 
were utilized in determining course grades, including homework, quizzes, two midterm 
exams, and a final exam. The first midterm covered material through topics dealing with 
summarizing data using tables and charts, descriptive statistics, and also the basics of 
probability. The second midterm covered material through probability distributions, 
sampling distributions, and confidence interval estimates for means and proportions. 

Total class size for each section was 46 students, although not all students completed all of 
the surveys and assessment tools we refer to here. For the control group there were 35 
participants and for the mapping group, 38 students completed all survey and assessment 
materials. These are thus the sample sizes for all that follows. The control group, as 
determined by coin flip, met at 9:30 AM and the group implementing concept maps met at 
2:00 PM, with each section meeting twice weekly for one hour and fifteen minutes. Of the 
participating students in each section, about 2/3 were sophomores and 1/3 juniors, with 
one freshman in the control group. Additionally, the control group was split nearly evenly 
between males and females, while the mapping section was 60% male. Both groups 
contained students who reported some previous exposure to statistics, either through 
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another course in statistics or as part of a math course. The control group had a greater 
proportion of students with no previous exposure, at 60% versus 43% for the mapping 
group. 

For the concept mapping section, development of concept maps was done at two points in 
the semester, in the class session just prior to each of the midterm exams. 

Implementation proceeded as follows. 

Concept Map Exercise 1 

Prior to the class session in which the first concept maps were developed students were 
given a short introduction to concept maps and instruction on their construction and use. 
This included reference to online resources at www.studvas.net/mappina for more 
information and examples. Following this short introduction to concept maps, students 
were informed that the next class period would be spent working in groups to develop such 
maps as a means of reviewing for the upcoming exam. Students were asked to bring in 
individual lists of terms and ideas from which they could draw in putting together their 
maps. The word around which the lists, and therefore the concept maps, were to be built 
was "population", representing the central unifying theme - study of a population 
characteristic of interest. The lists were meant to help students begin to start thinking 
about the connections between the topics thus far and prepare for discussion and 
collaboration with others in developing the concepts maps. 

On the day of implementation students were assigned to groups of 5 or 6. Each group was 
given scratch paper for developing rough drafts, colored markers, and a poster sized sheet 
of paper for the final product. Group members used their lists to develop a group concept 
map linking material covered up to that point and using the term "Population" as the 
central theme. This took place in three phases: first, groups were given about twenty-five 
minutes to brainstorm and come up with initial drafts of concept maps; next groups were 
asked to assign members to visit with other groups to observe their approaches and get 
ideas for improvement; finally, after 10 minutes of "mingling", the groups came back 
together to complete their group maps. In the last few minutes of class each of the eight 
groups presented their map to the class. 

Concept Map Exercise 2 

The second concept mapping exercise took place about a month after the first and, as a 
continuation of the initial concept mapping experience, was intended to expand upon the 
previously constructed maps. Again the unifying theme was the study of a population, 
only this time we added topics and ideas from probability and probability distributions, with 
the intended focus being on how sampling distributions fit in with everything else covered 
in the course up to that point. As in the previous exercise, students were instructed to 
bring in a list of terms and ideas, focusing on material covered since the first mapping 
exercise. 

On the day of the exercise, the following took place: 

- In an effort to assess the usefulness of concept mapping in aiding student 

understanding, a short quiz was given at the beginning of the class period, with a 
follow-up quiz given after the exercise. 
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- The maps from the previous exercise were posted in the front of the classroom so 
students could draw from other's ideas. 

- Students were assigned to eight new groups of 5 or 6 students. Groups were 
asked to start from scratch, utilizing the old maps and also the student's individual 
lists to develop concept maps linking all of the material covered in the course to 
that point. 

- The class was given 15 minutes for each group to come up with a first draft and 
then again allowed to mingle with other groups to see their approaches. 

- After 10 minutes of mingling, the groups came back together and spent 10 minutes 
completing their maps. 

- During the post-quiz the completed maps were posted in front of the room and 
students were asked to vote on the best mapping. Groups were barred from voting 
on their own. Extra credit points were given to the two groups with the most 
votes. 


Evidence of Effectiveness? 

To evaluate whether or not the objectives of the experiment have been met, comparisons 
are made between the section that produced concept maps and the control group who did 
not. Both goals stated above were assessed through the use of pre- and post-tests and 
surveys. 

Assessment of learning outcomes was done through the use of the Statistics Concept 
Inventory (SCI) (see the SCI website at https://enaineerina.purdue.edu/SCI/index.htm for 

details on this instrument and its use). This 38 question testing instrument is designed to 
measure the impact of alternative teaching and learning strategies on the understanding of 
core concepts typically covered in an introductory statistics course. This is done through 
comparison of results obtained by administering the test at the beginning of the course, 
prior to any instruction, and then again at the end of the course. Both the mapping group 
and the control group took the tests on the same days. 

Potential impact on student study habits and learning strategies over the semester was 
also tracked. This was done through the use of the GAMES Survey Instrument developed 
in Svinicki (2004). This survey, intended to reveal student learning styles and study 
methods, also was administered both at the beginning and at the end of the course in 
order to track how those may change over the semester. This part of the assessment is 
not course or content specific, but merely an attempt to see if students report changes in 
learning behavior over the semester and attempt to determine if exposure to concept 
mapping has any influence on this. 

In addition to the above, standard assessment tools -- exams, quizzes, and homework -- 
were used and scores on these were the primary determinant of student's course grades. 
In analyzing learning outcomes we use several bases for comparison, including pre and 
post results for the SCI, percentage point improvement on the SCI, normalized gain on the 
SCI, and overall course grade. Normalized gain is measured as 
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_ . PostSCI - PreSCI 

Gam = -, 

MaxSCI - PreSCI 

where MaxSCI = 38, the maximum possible correct. Gain is thus a measure of the 
proportion of the shortfall in points between the score on the pre-test SCI and the 
maximum possible score that is made up on the post-test (see Hake(2001) and references 
therein for a more complete discussion of this measure). Table 1 reports the means and 
standard deviations of the results from each of these measures for each group in the 

study. The average gain reported in the table is the average of the individual gains from 
each section. Alternatively, the normalized gain can be calculated based on the class 
average as a whole. This method generates a gain of 9.8% for the control group and 
7.2% for the concept mapping group. Although slightly larger in magnitude, the results 
are not qualitatively different. 


Table 1. Means and standard deviations of scores 




Section 




Control 

N = 35 

CM 

N = 38 

Total 

N = 73 

PREPct 

Mean 

s.d. 

39.5% 

9.7% 

39.3% 

12.3% 

39.4% 

11.1% 

PostPct 

Mean 

s.d. 

45.4% 

10.8% 

43.6% 

12.2% 

44.5% 

11.5% 

SCIJmprove 

Mean 

s.d. 

5.9% 

10.1% 

4.4% 

12.8% 

5.1% 

11.5% 

SCI_Gain 

Mean 

s.d. 

9.0% 

17.1% 

5.2% 

19.8% 

7.0% 

18.5% 

Course_Grade 

Mean 

s.d. 

82.0% 

11.2% 

84.7% 

8.3% 

83.4% 

9.8% 


Table 1 indicates that outcomes on the pre-test SCI were very close between the two 
sections, indicating a similar average starting point for both groups. We also see that 
outcomes were close, on average, for the remaining instruments as well, with average 
improvement and gain by the control group slightly greater on the SCI than that for the 
concept mapping group. Table 2 reports t -test results at the 5% level of significance for 
equality of means between the two sections for each of the outcomes reported in Table 1. 
These were done under the pre-tested assumption of equivalence in variances across 
groups. In no case can we reject the null hypothesis that average class performance is the 
same across the two groups. Unfortunately, this provides no evidence that use of concept 
maps made any difference in performance on post-tests or course grades. 


Table 2. t-test results for difference in means 
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PREPct 

PostPct 

SCIJmprove 

SCI_Gain 

Course_Grade 

Mean Difference 

0.21% 

1.78% 

1.58% 

3.73% 

-2.70% 

Std. Error Difference 

2.61% 

2.71% 

2.72% 

4.35% 

2.30% 

t 

0.0790 

0.6590 

0.5800 

0.8580 

-1.1770 

df 

71 

71 

71 

71 

71 

Sig. (2-tailed) 

0.937 

0.512 

0.564 

0.394 

0.243 

95% Cl for Difference: 






Lower 

-5.01% 

-3.62% 

-3.84% 

-4.94% 

-7.28% 

Upper 

5.42% 

7.19% 

7.00% 

12.41% 

1.88% 


Using additional data on student characteristics that was collected with the SCI 
(summarized in Table 3), we can control for the effects of gender, class standing, and 
whether or not students have had previous exposure to statistics, say in high school or in a 
math class at some point. Table 4 reports univariate ANOVA analysis of the results for 
pre-test SCI scores, post-test SCI scores, improvement on the SCI, normalized gain, and 
final course grades. Along with class section, the new variables are included as factors in a 
multifactor univariate ANOVA model. In addition to allowing us to assess possible 
interactions of these additional factors with class section we may also gain insight into 
potential effects on performance outcomes across these subcategories as well. In addition 
to p-values for the significance of each factor or interaction effect in the model. Table 4 
reports the partial eta squared statistic, an indicator of effect size of each on variability in 
the dependent variable. Finally, the R 2 and adjusted R 2 measures of fit for each model are 
reported. 


Table 3. Additional factors 


Between-Subjects Factors 

N 

Section 

Control 

35 


CM 

38 

Gender 

Male 

41 


Female 

32 


Sophomore 

52 

Standing 

Junior 

20 


Freshman 

1 


None 

39 

StatEx 

Some Previous 
Experience 

34 


Even allowing for the possibility of interaction effects, section now only directly has a 
statistically significant impact on overall course grade, with the concept mapping group 
averaging 84.7% for the course and the control group averaging 82.0%. There is also a 
significant interaction effect with class standing on course grade, with sophomores in the 
control section outperforming juniors to a much greater degree than in the concept 
mapping section. The effect size of the interaction term is larger than that for section 
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itself, so it is likely that this is the driving force behind section being significant at all. Thus 
even here there is little convincing evidence that the concept mapping exercises had an 
effect on course grade. All other interaction terms involving section were insignificant. 

Class standing and previous statistics experience, which seems to have hurt rather than 
helped, were both significant and individually had much larger effect sizes on course grade 
than did class section. These variables, along with gender, all have significant interaction 
effects on course grade as well. 
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Table 4. Univariate ANOVA Results 


Dependent Variable: 

PREPct 

R 2 = .325 (Adj. R 2 = .132) 

PostPct 

R 2 = .381 (Adj. R 2 = .204) 

SCIJmprove 

R 2 = .254 (Adj. R 2 = .041) 

SCI_Gain 

R 2 = .293 (Adj. R 2 = .092) 

Course_Grade 

R 2 = .456 (Adj. R 2 = .300) 

Source 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Corrected Model 

0.077 

0.325 

0.018 

0.381 

0.303 

0.254 

0.151 

0.293 

0.001 

0.456 

Intercept 

0.000 

0.849 

0.000 

0.883 

0.016 

0.099 

0.031 

0.080 

0.000 

0.977 

Section 

0.959 

0.000 

0.318 

0.018 

0.340 

0.016 

0.346 

0.016 

0.023 

0.088 

Gender 

0.344 

0.016 

0.001 

0.176 

0.027 

0.085 

0.025 

0.087 

0.045 

0.070 

Standing 

0.076 

0.088 

0.006 

0.166 

0.083 

0.085 

0.050 

0.101 

0.000 

0.270 

StatEx 

0.834 

0.001 

0.245 

0.024 

0.213 

0.028 

0.103 

0.047 

0.002 

0.152 

Section * Gender 

0.959 

0.000 

0.171 

0.033 

0.229 

0.026 

0.096 

0.049 

0.663 

0.003 

Section * Standing 

0.195 

0.030 

0.333 

0.017 

0.041 

0.072 

0.025 

0.086 

0.012 

0.106 

Gender * Standing 

0.827 

0.001 

0.022 

0.090 

0.023 

0.089 

0.015 

0.100 

0.077 

0.055 

Section * Gender * 
Standing 

0.057 

0.063 

0.024 

0.087 

0.740 

0.002 

0.628 

0.004 

0.213 

0.028 

Section * StatEx 

0.279 

0.021 

0.558 

0.006 

0.647 

0.004 

0.503 

0.008 

0.818 

0.001 

Gender * StatEx 

0.045 

0.070 

0.759 

0.002 

0.035 

0.077 

0.024 

0.088 

0.071 

0.057 

Section * Gender * 

StatEx 

0.280 

0.021 

0.743 

0.002 

0.490 

0.009 

0.337 

0.016 

0.215 

0.027 

Standing * StatEx 

0.025 

0.087 

0.120 

0.043 

0.505 

0.008 

0.521 

0.007 

0.002 

0.165 

Section * Standing * 
StatEx 

0.966 

0.000 

0.308 

0.019 

0.374 

0.014 

0.181 

0.032 

0.846 

0.001 

Gender * Standing * 

StatEx 

0.801 

0.001 

0.760 

0.002 

0.612 

0.005 

0.350 

0.016 

0.616 

0.005 

Section * Gender * 
Standing * StatEx 

0.159 

0.035 

0.751 

0.002 

0.315 

0.018 

0.292 

0.020 

0.682 

0.003 


Shaded cells indicate significance of factors or interactions at a 10% level or less. 


The only other statistically significant impacts from section appear to be in interaction with 
other variables. For the pre-test SCI score, section interacts with class standing and 
gender to produce a relatively small and likely inconsequential effect, given the R 2 of the 
model. Section interacts with class standing to have a small significant effect on absolute 
improvement in SCI scores between the pre- and post-tests, with sophomores in the 
control section again significantly outperforming juniors, with 7.6 and 0.9 percentage point 
gains, respectively. Finally, the normalized gain on the SCI had significant interaction 
effects between section and gender and section and standing. In both cases inspection of 
cross tabulations indicates this is due to significant differences within the control section, 
rather than the mapping section. Males in the control section averaged a normalized gain 
of 12.1 percent and females 4.3 percent, whereas in the concept mapping section the 
gains were 6.1 percent and 4.3 percent, respectively. At any rate, the effect sizes of these 
interactions are again quite small, especially given the explanatory power of the overall 
models. 
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Unfortunately, the results thus far offer little evidence that concept mapping had any 
impact on course outcomes or on student's conceptual understanding of statistics, as 
measured by the SCI. There may be a number of explanations for this, not the least of 
which is the small sample size upon which we are basing our conclusions. Additionally, 
any gains in understanding that may be realizable from the incorporation of concept maps 
into such a course may not be well measured by either the SCI or standard course 
assessment tools, as attempted here. This bears further consideration in future 
experiments of this sort. 


GAMES Self-Reported Outcomes 

Although the results pertaining to course specific learning outcomes were somewhat 
disappointing, it is possible that the introduction of a learning tool such as concept maps 
can have an impact on how students approach learning. The GAMES survey, described 
above, was administered in order to help assess whether or not this was the case. For 
each learning strategy listed on the survey form, students provide a ranking on a scale 
from one to five on their use of that particular strategy, with 1 indicating the strategy is 
never used, 2 rarely used, 3 sometimes, 4 often, and 5 indicating the technique is always 
used. 

Table 4 contains the questions from the GAMES survey and reports the proportion of each 
class section reporting an increase in the use of each particular learning strategy over the 
course of the semester. Here we employ two alternative definitions of what an increase in 
usage may pertain to. We define an increase of Type A as representing greater usage of a 
skill on the post-survey than reported on the pre-survey for that item, so this measure 
counts any upward movement in reported usage as an increase. A Type B increase, on the 
other hand, is measured as a movement from a report of either "Never" or "Seldom" on 
the pre-survey to any of the higher three rankings on the post-survey, representing an 
adoption of the method by a student who reported little, if any, use of it prior to the 
course. Any differences between sections that are statistically significant at the 10% level 
are indicated by an asterisk in the CM column. We do note that in a number of cases there 
were no changes in a student's use of a particular strategy and some even reported 
declines in usage of some strategies from what was initially reported. The shaded 
questions in the table are those which refer specifically to the use of concept maps or of 
making explicit connections between topics, which is a primary characteristic of concept 
maps. 

The results in Table 4 indicate that the concept mapping group reported greater 
proportionate increases in learning strategy usage over the semester than did their 
counterparts in the control group for slightly more than half of the surveyed strategies 
(56%). They also report a greater adoption rate for previously unused/seldom used 
learning strategies for 65% of the techniques covered. Of course the reported differences 
in proportion are statistically significant for only a handful of the survey questions, due to 
the small sample sizes. 
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Table 4. Proportion of class reporting an increase in the use of each learning strategy 


Type of increase (see text for definitions) : 

Type A 

Type B 

Section: 

Control 

CM 

Control 

CM 

Goal-oriented study 





1. Analyze what I have to do before beginning to study. 

28.6% 

26.3% 

5.7% 

7.9% 

2. Set a specific content learning goal before beginning to study. 

34.3% 

44.7% 

14.3% 

23.7% 

3. Set a specific work elfort (time or amount) before beginning to study. 

31.4% 

26.3% 

17.1% 

18.4% 

4. Figure out why I am learning the material fm about to study. 

45.7% 

42.1% 

14.3% 

23.7% 

5. Be sure to understand what is expected of me in terms of learning and assignments. 

25.7% 

21.1% 

0.0% 

5.3% 

Active Study 





6. Make notes in the maigins of the text when I read. 

28.6% 

42.1% 

17.1% 

21.1% 

7. Ask myself questions before, during, and after studying. 

40.0% 

42.1% 

20.0% 

13.2% 

8. Pause periodically to summarize or paraphrase what I've just studied. 

28.6% 

34.2% 

11.4% 

5.3% 

9. Create outlines, concept maps, or organizational charts of how the ideas fit together. 

31.4% 

50.0% * 

14.3% 

28.9% * 

10. Look for connections between what fm studying right now and what I've studied in 
the past or heard in class. 

31.4% 

34.2% 

8.6% 

18.4% 

11. Write down questions I want to ask the instructor. 

34.3% 

50.0% * 

17.1% 

28.9% 

12. Reotganize and fill in the notes I took in class. 

34.3% 

31.6% 

14.3% 

13.2% 

13. Work through any problems that are illustrated in the text or in my class notes. 

28.6% 

36.8% 

17.1% 

18.4% 

14. Create vocabulary lists with definitions and my own examples. 

20.0% 

28.9% 

8.6% 

7.9% 

15. Take breaks periodically to keep from getting too tired. 

17.1% 

10.5% 

0.0% 

0.0% 

Meaningful and memorable 





16. Make up my own examples for concepts I am learning. 

20.0% 

18.4% 

11.4% 

10.5% 

17. Put things in my own words. 

8.6% 

21.1% * 

0.0% 

5.3% 

18. Make vivid images of concepts and relationships among them. 

31.4% 

13.2% * 

8.6% 

5.3% 

19. Make connections between what I am studying and past classes or units. 

28.6% 

18.4% 

8.6% 

5.3% 

20. Be sure I understand any example the instructor gave me. 

31.4% 

31.6% 

8.6% 

10.5% 

21. Create concept maps and diagrams that show relationships among concepts. 

34.3% 

39.5% 

17.1% 

28.9% 

22. Ask the instructor for more concrete examples and picture them in my mind. 

22.9% 

36.8% * 

11.4% 

23.7% * 

23. Look for practical applications and real life settings for the things fm learning. 

14.3% 

15.8% 

8.6% 

10.5% 

Explain to understand 





24. After studying, meet with a partner to trade questions and explanations. 

34.3% 

52.6% * 

14.3% 

31.6% * 

25. Write out my own descriptions of the main concepts. 

28.6% 

23.7% 

11.4% 

15.8% 

26. Discuss the course content with anyone willing to listen. 

17.1% 

34.2% * 

8.6% 

23.7% * 

27. Answer questions in class. 

34.3% 

26.3% 

22.9% 

21.1% 

28. Make a class presentation. 

25.7% 

18.4% 

8.6% 

10.5% 

29. Help another student who is behind in progress. 

22.9% 

34.2% 

11.4% 

10.5% 

Self-monitor 





30. Make sure I can answer my own questions during studying. 

34.3% 

26.3% 

11.4% 

5.3% 

31. Work with another student to quiz each other on main ideas. 

25.7% 

34.2% 

11.4% 

13.2% 

32. Keep track of things I don't understand and note when they finally become clear and 
what made that happen. 

31.4% 

26.3% 

17.1% 

10.5% 

33. Have a range of strategies for learning so that if one isn't working I can try another. 

34.3% 

42.1% 

20.0% 

28.9% 

34. Remain aware of mood and energy levels during study and respond appropriately if 
either gets problematic. 

37.1% 

36.8% 

14.3% 

10.5% 


^indicates difference is significant at 10% level 
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Of the categories in the survey, the concept mapping group shows greatest relative gains 
of type A in the "Active Study" and "Meaningful and Memorable" categories. Within Active 
Study, questions 9 and 10 directly pertain to learning strategies that may be related to the 
use of concept mapping. For both of these strategies the mapping group reported a 
greater percentage increase in use than did the control group. However, only for question 
9, which mentions concept maps explicitly, is the difference statistically significant. Within 
the Meaningful and Memorable category two questions pertaining directly to concept 
mapping's potential strengths, questions 18 and 19, go the other direction, with the 
control group exhibiting a proportionately greater increase in the use of these strategies, 
with the difference in question 18 being statistically significant. Question 21, again directly 
mentioning concept maps, shows only slightly greater increases over the control group for 
those actually experiencing their use in the course, despite the results for question 9, 
which is very similar. Using this measure, there is only mixed evidence of any positive 
impact on learning behavior and study methods from the introduction of concept maps. Of 
the two statistically significant questions that pertain directly to concept mapping, the 
evidence is contradictory. 

For the Type B increases, which essentially measure new adoption of the learning strategy 
in question, there were even fewer statistically significant differences in relative take up 
rates between the two groups. Of these, question 9 in the Active Study category indicates 
the concept mapping group had double the adoption rate for this strategy than that of the 
control group. Question 21 also indicates a large, but statistically insignificant difference 
here. This may seem an obvious result of the introduction of these methods in that 
section, but the second mapping exercise took place about a month previous to the post¬ 
survey, with no mention of it in between. This leads us to believe that student reporting of 
concept maps being adopted as a learning strategy is genuine. If that is the case, and 
students did add this learning tool to their repertoire, then that can only be considered a 
success. 

Another use for the data gleaned from the GAMES surveys is to see if increased use or 
adoption of the learning strategies addressed on it had any impact on the learning 
outcomes reported above. We investigate this for those questions, highlighted in table 4, 
that pertain most to the use of concept mapping, the main aim of this study. Table 5 and 
Table 6 report the results of multifactor univariate ANOVA for the impacts on improvement 
in SCI scores, the normalized gain in SCI scores, and overall course grades from gains of 
type A and type B in the highlighted survey questions. For this analysis the data was 
pooled across sections and the analysis done separately for each GAMES category, due to 
overlap in the questions being asked. 
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Table 5. Impact of Type A increases in GAMES strategies on learning outcomes 


Dependent Variable: 

SCIJmprovePCT 

SCI_Gain 

Course_Grade 

Source 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Corrected Model 

0.383 

0.043 

0.520 

0.032 

0.023 

0.128 

Intercept 

0.000 

0.198 

0.001 

0.148 

0.000 

0.985 

TypeA_9 

0.562 

0.005 

0.635 

0.003 

0.006 

0.104 

Type A_10 

0.141 

0.031 

0.152 

0.029 

0.044 

0.058 

TypeA_9 * TypeA_10 

0.337 

0.013 

0.646 

0.003 

0.152 

0.030 


R Squared = .043 
(Adj R Squared = .001) 

R Squared = .032 
(Adj R Squared = -.010) 

R Squared = .128 (Adj 

R Squared = .090) 

Dependent Variable: 

SCIJmprovePCT 

SCI_Gain 

Course_Grade 

Source 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Corrected Model 

0.229 

0.129 

0.294 

0.118 

0.031 

0.204 

Intercept 

0.030 

0.071 

0.084 

0.045 

0.000 

0.978 

Type A_18 

0.051 

0.057 

0.076 

0.048 

0.549 

0.006 

Type A_19 

0.764 

0.001 

0.900 

0.000 

0.273 

0.018 

TypeA_21 

0.228 

0.022 

0.225 

0.023 

0.092 

0.043 

TypeA_18 *TypeA_19 

0.379 

0.012 

0.576 

0.005 

0.758 

0.001 

Type A_18 * Type A 21 

0.379 

0.012 

0.368 

0.012 

0.384 

0.012 

TypeA_19 * TypeA_21 

0.986 

0.000 

0.929 

0.000 

0.004 

0.118 

TypeA_18 *TypeA_19 * 
TypeA 21 

0.288 

0.017 

0.263 

0.019 

0.694 

0.002 


R Squared = .129 
(Adj R Squared = .036) 

R Squared =.118 
(Adj R Squared = .023) 

R Squared = .204 
(Adj R Squared = .119) 


Shaded cells indicate significance at a 10% level 


Table 5 indicates there is weak evidence that greater usage of strategies 9 and 10 on the 
GAMES survey had an impact on overall course grades, but no evidence that these 
mattered for performance on the SCI. Questions 10 and 19 are nearly identical, yet in the 
lower panel of the table question 19 only has an impact in combination with the use of 
concept maps, as indicated by question 21. Questions 9 and 21 also are very close, and 
both provide weak evidence that greater use of concept maps or similar tools impacted 
course grades. The only evidence of effect on the SCI is for improvement of the use of 
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visualizing concepts and relationships from question 18. 
at best. 


Here again the evidence is weak 


Table 6. Impact of Type B increases in GAMES strategies on learning outcomes 


Dependent Variable: 

SCIJmprovePCT 

SCI_Gain 

Course_Grade 

Source 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Corrected Model 

0.110 

0.083 

0.175 

0.069 

0.407 

0.041 

Intercept 

0.003 

0.124 

0.011 

0.089 

0.000 

0.966 

Type B 9 

0.470 

0.008 

0.484 

0.007 

0.286 

0.016 

TypeB_10 

0.718 

0.002 

0.859 

0.000 

0.312 

0.015 

TypeB_9 *TypeB_10 

0.056 

0.052 

0.071 

0.046 

0.589 

0.004 


R Squared = .083 
(Adj R Squared = .043) 

R Squared = .069 
(Adj R Squared = .028) 

R Squared = .041 
(Adj R Squared = -.001) 

Dependent Variable: 

SCIJmprovePCT 

SCI_Gain 

Course_Grade 

Source 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Sig. 

Partial Eta 
Squared 

Corrected Model 

0.107 

0.124 

0.127 

0.118 

0.152 

0.111 

Intercept 

0.414 

0.010 

0.454 

0.008 

0.000 

0.921 

TypeB 18 

0.743 

0.002 

0.986 

0.000 

0.264 

0.019 

TypeB 19 

0.379 

0.012 

0.399 

0.011 

0.772 

0.001 

TypeB_21 

0.015 

0.084 

0.011 

0.092 

0.277 

0.018 

TypeB_18 *TypeB_19 


0.000 


0.000 


0.000 

TypeB_18 * TypeB_21 

0.117 

0.036 

0.109 

0.038 

0.800 

0.001 

TypeB_19 * TypeB_21 

0.028 

0.070 

0.034 

0.066 

0.032 

0.067 

TypeB 18 * TypeB 19 * 
TypeB_21 


0.000 


0.000 


0.000 


R Squared = .124 
(Adj R Squared = .058) 

R Squared =.118 
(Adj R Squared = .052) 

R Squared =.111 
(Adj R Squared = .045) 


Shaded cells indicate significance at a 10% level 


Table 6 does provide some evidence that the adoption of the selected learning strategies 
had some impact on SCI performance. The combination of making connections between 
topics and formalizing those connections in some way, as indicated in questions 9 and 10 
on the survey, does show a slight statistically significant impact on improvement on the 
SCI over the semester, as measured by the normalized gain and the percentage point 
change in scores. Question 21 addresses the use of concept maps specifically and the 
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evidence her shows that adoption of this learning tool also had a small but significant 
effect on the SCI measures. 

All in all there is not a lot of evidence here suggesting that there was a great benefit from 
the use of concept maps for either the students that participated in the exercises or for all 
of the students that reported greater usage or adoption of this type of tool on the GAMES 
survey. While this is a disappointing result, the presence of even weak evidence of effect 
suggests it is worthwhile to continue to investigate these questions and collect additional 
data in the hopes of strengthening the results. 


What Should Be Done Next Based On What We Learned? 

Heinze, Fry and Novak (1990) noted that the introduction of concept maps should not be 
considered a "quick fix." Concept mapping does appear to move students toward 
meaningful learning but this movement may be slow. This may be due to the fact that it 
takes time for students to learn the process of concept mapping and realize its potential 
(Wandersee, 1990). Or, as suggested by Santhanam, et. al. (1998) students, especially 
underclassmen, view rote memorization as the optimum approach to study. These habits 
that have served the students well throughout high school may be difficult to break. 
Although in our case it appears that the concept mapping did change study behaviors as 
reflected by the GAMES® survey, it is probably the case that the mapping was done too 
infrequently to have a significant impact on performance. 

One issue raised in the literature has been does it matter who prepares the map and how. 
On the one hand concept maps can be constructed by the teacher, by students in groups 
or by individual students. Students can fill in the details of a map that has been partially 
constructed by the instructor or can develop a map from scratch. Wandersee (1990) 
argued that the educational benefit accrues chiefly to the mapper. As noted earlier 
according to previous studies mapping done collaboratively enhances performance more 
than that done by individuals. However, given the independent effects of collaboration on 
performance we cannot be sure how much of the improvement should be attributed to 
concept mapping. If Wandersee is correct, there is an argument to be made for individual 
students to construct their own maps. One strategy that might be considered is for each 
student to prepare a concept map rather than simply a list of concepts prior to the exercise 
and then have students work from those maps to construct one group map. In this way 
we can retain the benefits of collaboration while increasing individual accountability. 

Finally, no attempt was made to grade the concept map itself. Several scoring rubrics can 
be found in the literature with the common feature of scoring on the basis of complexity 
and validity. For instance, scores are based on the number of concepts and correctness of 
relationships indicated by the cross links. Making the concept mapping a graded exercise 
likely would increase student motivation. 


Conclusion 

The goal of this exercise was to determine the effectiveness of a particular learned focused 
activity. While much previous work suggests that such activities are useful in enhancing 
student learning, there is typically little empirical support offered. In this paper we find 
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little evidence to support concept mapping as an effective tool given the measurement 
tools employed. One possible explanation for this involves the types of outcomes 
measured, which included pre- and post-tests and course grades. These gross measures 
of performance may not be sensitive enough to the type of learning that concept mapping 
enhances, i.e. the relationships between the concepts covered in the course that may not 
be directly addressed on a multiple choice exam. It may also be the case that the design 
and frequency of the activity was insufficient to measurably alter learning outcomes in the 
course. However, the self-reported results from students on the GAMES survey suggested 
some potential for improvement in study habits as a result of introducing students to 
concept mapping. 
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